Method and apparatus for calibration of camera system, and method of manufacturing camera system

ABSTRACT

A first mirror and a second mirror are disposed in such a manner that their reflective surfaces are parallel and face toward each other. A mark is drawn between the first mirror and the second mirror. An image of the reflections of the mark in the first mirror and the second mirror is captured with a camera. Calibration of the camera system is performed based on the image.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application is a Divisional Application of, and claims the benefitof priority under 35 U.S.C. § 120 from, U.S. application Ser. No.10/704,550, filed Nov. 12, 2003 now U.S. Pat. No. 7,124,046 and claimsthe benefit of priority under 35 U.S.C. § 119 from Japanese PatentApplications No. 2002-346914 filed on Nov. 29, 2002 and No. 2003-324615filed on Sep. 17, 2003. The entire contents of each of the aboveapplications are incorporated herein by reference.

BACKGROUND OF THE INVENTION

1) Field of the Invention

The present invention relates to a method and an apparatus forcalibration of a camera system, and a method of manufacturing the camerasystem.

2) Description of the Related Art

Recently, a camera system that recognizes information, for example,position information, of an object in an image taken by a camera hasbeen developed.

For instance, the technology that recognizes an obstacle from a picturetaken by two or more cameras is proposed in Japanese Patent ApplicationLaid-open Publication No. 2001-243456.

Such a camera system is sometimes installed on the vehicles. A camera,which may be installed inside or outside the vehicle, captures an imageof a forward area of the vehicle. It is recognized from the imagewhether an obstacle (e.g. other vehicles) is present in front of thevehicle. If an obstacle is present, a distance between the obstacle andthe vehicle is calculated. This camera system is calibrated immediatelyprior to its shipment. The calibration process includes calibration ofthe installation position of the camera as well as setting of variousparameters that are required for taking a picture.

The calibration is performed according to the following procedure: 1)fix a camera at a predetermined position so that the camera covers aregion in which two or more marks are placed; 2) take a picture of aregion and obtain position information of the marks (hereinafter,“picture mark information”) in the picture; and 3) perform calibrationprocess based on a difference between the picture mark information andactual position information of the marks (hereinafter, “actual markinformation”), which is know in advance. The calibration can beperformed more accurately if the region covered by the camera is largeand when there are lot of marks in the region. In other words, forbetter accuracy, a large space (e.g. distances of about several tens toa hundred meters from the camera) and a lot of marks are required.

Other approach is to use the Planar Projection Stereopsis. A camerasystem that employs the planar projection stereopsis is disclosed in,for example, Japanese Patent Application Laid-open Publication No.2001-243456. In this camera system, since it is necessary to obtainparameters for plane projection, its calibration also requiresinformation that indicates how the plane is projected on the picture. Inother words, for better accuracy, a large plane and a lot of marks arerequired.

Thus, the conventional camera systems give better accuracy only if alarge space is available. This requirement makes it impossible toperform the calibration inside the factory where the camera system ismanufactured. As a result, to perform the calibration, the camera systemis installed on a vehicle and the vehicle is carried to a place such asa road, which is outside the factory, where a lot of marks are placed.However, this makes the process cumbersome.

SUMMARY OF THE INVENTION

It is an object of the present invention to at least solve the problemsin the conventional technology.

A method for calibration of a camera system according to one aspect ofthe present invention includes disposing a first mirror and a secondmirror in such a manner that reflective surfaces of the first mirror andthe second mirror are parallel and face toward each other; placing amark between the first mirror and the second mirror; capturing with thecamera an image of the marks reflected in the first mirror and thesecond mirror; and performing calibration of the camera system based onthe image.

A method for calibration of a stereo camera system that includes aplurality of cameras according to another aspect of the presentinvention includes disposing a first mirror and a second mirror in sucha manner that reflective surfaces of the first mirror and the secondmirror are parallel and face toward each other; placing at least twomarks between the first mirror and the second mirror; capturing with thecameras an image of the marks reflected in the first mirror and thesecond mirror; determining position of the marks in each of the images,wherein the position of each of the marks is represented by a pair ofcoordinates; and calculating projective transformation parametersbetween the images from the position of the marks and by using as areference a plane where the marks are placed.

An apparatus for calibration of a camera system according to stillanother aspect of the present invention includes a first mirror having areflective surface; a second mirror that is parallel to the first mirrorand having a reflective surface, wherein the reflective surfaces of thefirst mirror and the second mirror face toward each other; and a markthat is disposed between the first mirror and the second mirror.

An apparatus for calibration of a stereo camera system that includes aplurality of cameras according to still another aspect of the presentinvention includes a first mirror having a reflective surface; a secondmirror that is parallel to the first mirror and having a reflectivesurface, wherein the reflective surfaces of the first mirror and thesecond mirror face toward each other; a plurality of marks that aredisposed between the first mirror and the second mirror; and an imageprocessor that calculates projective transformation parameters betweenimages captured by the cameras from a position of the marks in theimages and by using as a reference a plane where the marks are placed,wherein the position of each of the marks is represented by a pair ofcoordinates.

A method of manufacturing a camera system according to still anotheraspect of the present invention includes installing the camera in apredetermined position; disposing a first mirror and a second mirror insuch a manner that reflective surfaces of the first mirror and thesecond mirror are parallel and face toward each other; placing a markbetween the first mirror and the second mirror; capturing with thecamera an image of the marks reflected in the first mirror and thesecond mirror; and performing calibration of the camera system based onthe image.

A method of manufacturing a stereo camera system, which includes aplurality of cameras and a memory, according to still another aspect ofthe present invention includes installing the cameras in a predeterminedposition; disposing a first mirror and a second mirror in such a mannerthat reflective surfaces of the first mirror and the second mirror areparallel and face toward each other; placing at least two marks betweenthe first mirror and the second mirror; capturing with the cameras animage of the marks reflected in the first mirror and the second mirror;determining position of the marks in each of the images, wherein theposition of each of the marks is represented by a pair of coordinates;calculating projective transformation parameters between the images fromthe position of the marks and by using as a reference a plane where themarks are placed; and storing the projective transformation parametersin the memory.

These and other objects, features and advantages of the presentinvention are specifically set forth in or will become apparent from thefollowing detailed descriptions of the invention when read inconjunction with the accompanying drawings.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a schematic view of a calibration apparatus according to afirst embodiment of the present invention;

FIG. 2 is a schematic top view of the calibration apparatus;

FIG. 3 is an illustration indicating a lot of reflective images of amark;

FIG. 4 is a flow chart of a calibration method according to the firstembodiment;

FIG. 5 is an example of an image of the mark arranged in a linediagonally across the mirror;

FIG. 6 is a schematic top view of a calibration apparatus according to asecond embodiment of the present invention;

FIG. 7 is a flow chart of a calibration method according to the secondembodiment;

FIG. 8 is an example of an image captured by one of two cameras;

FIG. 9 is an example of an image captured by the other of the twocamera;

FIG. 10 is a schematic view of a calibration support device according toa third embodiment of the present invention;

FIG. 11 is an illustration indicating a relation between an optical axisof the camera and an upper bound line to obtain the reflective image inthe first embodiment;

FIG. 12 is a schematic view of a calibration support device according toa fourth embodiment of the present invention;

FIG. 13 is an illustration indicating a relation between the opticalaxis of the camera and the upper bound line to obtain the reflectedimages when the calibration support device is used for the calibration;

FIG. 14 is a flow chart of a method of manufacturing the camera system;

FIG. 15 is a modification of the calibration support device;

FIG. 16 is another modification of the calibration support device; and

FIG. 17 is still another modification of the calibration support device.

DETAILED DESCRIPTION

Exemplary embodiments of a method and an apparatus for calibration of acamera system, and a method of manufacturing the camera system relatingto the present invention will be explained in detail below withreference to the accompanying drawings.

FIG. 1 and FIG. 2 are schematic views of a calibration apparatusaccording to a first embodiment of the present invention. Thecalibration apparatus 100 includes a calibration support device 110, acamera 105, and an image processor 106. The calibration support device110 includes a mirror 101 (a first mirror), a mirror 102 (a secondmirror), and a plate-like base 103. The mirrors 101 and 102 are disposedon the base 103 in such a manner that the mirrors 101 and 102 are normalto the base 103, there is a predetermined distance L between the mirrors101 and 102, and the reflective surfaces of the mirrors 101 and 102 faceeach other.

It is preferable that the distance between the mirrors 101 and 102 is 1meter, however, the distance may be longer or shorter than 1 meter. Amark 104 is placed on the base 103 between the mirrors 101 and 102. Itis preferable that the mark 104 is easily distinguishable from any otherobject. One approach is to make the mark 104 from a light source (e.g.,a light, an LED, a lamp, or a candle). Another approach is to form themark with a paint having high-brightness (e.g., white paints orfluorescent paints).

The calibration support device 110 and the camera 105 are disposed inthe manner as shown in FIGS. 1 and 2. The image processor 106 is, forexample, a computer device such as a personal computer, and includes adisplay unit (e.g., a liquid crystal display or a cathode ray tube), aninput device (e.g., a mouse or a keyboard), and a CPU, a memory, and anexternal storage. This image processor 106 executes the calibrationprocessing according to programs stored in the external storage.

In the first embodiment, a situation that a lot of marks are scatteredover a vast area is simulated by the calibration support device 110 andthe camera.

Precisely, since the reflective surfaces of the mirrors 101 and 102 areparallel and face each other, when viewed from the direction of thecamera 105, as shown in FIG. 3, it appears as if a lot of marks 1041,1042, 1043, 1044, 1045 and so on, are arranged in a line over a longdistance, because, the mark 104 is reflected repeatedly.

Considering the position of the real mark 104 to be the origin, virtualmarks 1041, 1042, 1043 are obtained at distances A, B, C, respectively,from the origin such that C>B>A. If the mark 104 is equidistant from themirrors 101 and 102, then B=2×A and C=3×A so that the virtual marks1041, 1042, 1043 appear at the distances A, 2×A, 3×A from the origin. Onthe other hand, if the mark 104 is not equidistant from the mirrors 101and 102, then there may not be a specific relation between the distancesA, B, C.

In this arrangement, assuming that the distances between the mark 104and the mirrors 101 and 102, respectively, are known, the distancebetween the origin and the virtual marks can be calculated easily. Thecalculation is easier if the mark 104 is equidistant from the mirrors101 and 102, because, there is a specific relationship between thedistance between the origin and the virtual marks.

One approach is to place the camera 105 between the mirrors 101 and 102.Although this arrangement will output maximum number of virtual images,this arrangement is unrealistic. A better approach is to focus thecamera 105 from little above the mirror 102 (or the mirror 101) themirror 101 (or the mirror 102).

The flow chart in FIG. 4 details the calibration process. The mirrors101 and 102 are placed on the base 103 (step Sa1). In other words, themirrors 101 and 102 are disposed on the base 103 in such a manner thatthe mirrors 101 and 102 are normal to the base 103, there is apredetermined distance L between the mirrors 101 and 102, and thereflective surfaces of the mirrors 101 and 102 face each other.

A mark is placed on the base 103 between the mirrors 101 and 102 (stepSa2). Thus, the steps Sa1 and Sa2 realize the calibration support device110. If the calibration support device 110 is available, the steps Sa1and Sa2 are can be omitted. In other words, if the calibration supportdevice 110 is available, adjusting the positions of the mirrors, whichis a complex process, can be omitted.

During repeated use the mirrors 101 and 102 may shift relatively. If themirrors 101 and 102 shift relatively, it is necessary to adjust thepositions of the mirrors 101 and 102. The best positions of the mirrors101 and 102 are such that the mark 104 and a lot of virtual marks appearin a line.

When positioning the camera 105 and the calibration support device 110,the calibration support device 110 is fixed and the camera 105 is movednear the calibration support device 110. On the other hand, if, forexample, the camera 105 is already fixed to the vehicle, the calibrationsupport device 110 may be moved near the camera 105. When the camera 105is fixed, as shown in FIG. 2, so that the camera 105 points in adirection (i.e., the optical axis of the camera) other than thedirection of a straight line that joins the camera 105 and the mark 104(hereinafter, “a first arrangement”), the camera 105 captures an imagethat includes a lot of marks positioned on an inclined straight line.FIG. 5 is an example of such an image.

When the camera 105 is fixed so that the camera 105 points toward themark 104 (hereinafter, “a second arrangement”), the camera 105 capturesan image that includes a lot of marks positioned on a vertical straightline. Whether to employ the first arrangement or the second arrangementdepends on the purpose of the camera system. For example, if detectionof a distance between the vehicle and a lane line (a mark, e.g., a whiteline) is the purpose, it is preferable that the camera 105 does notpoint toward the mark (i.e., the lane line). In other words, in thiscase it is preferable to employ the first arrangement.

Once the positions between the camera 105 and the calibration supportdevice 110 are adjusted, the camera 105 takes a picture. As a result,the camera 105 captures an image containing a lot of marks (step Sa3).The image obtained by the camera 105 is input to the image processor106.

The image processor 106 performs calibration processes of settingvarious parameters or table data according to operator's instructions(step Sa4). This calibration processes includes setting of parametersfor image correction and setting of table data for distance detection.In the setting of parameters for image correction, when there are imagedistortions due to the characteristics of the camera 105 in the image,the image processor 106 acquires parameters for correcting thedistortions, and stores the parameters in a parameter setting memory ofthe camera system. When there are no image distortions due to thecharacteristics of the camera 105, an image that includes a lot of marksin a line is obtained.

On the other hand, when there are image distortions due to thecharacteristics of the camera 105, an image that includes a lot of markson a curve is obtained. The image processor 106 compares, when thecamera has the distortion characteristics, each of original positionswhere a lot of marks are arranged in the captured image withoutdistortion, with each of distorted positions where a lot of marks arearranged in the captured image with distortion.

The image processor 106 also calculates table data for distancedetection from the positions of the marks in the captured image andactual positions of the marks 104 and their virtual marks, and registersin a distance detection table. Respective distances from actual marks(the mark 104 and their virtual marks) corresponding to marks in thecaptured image to the camera 105 are calculated in advance (see FIG. 3).Therefore, the table data for distance detection can be created byassociating the positions of actual marks with the positions of themarks in the captured image.

In the table data, for example, a captured mark indicated as a distanceof XX (milimeters) from the bottom of the captured image is associatedwith an actual mark indicated as a distance YY (meters) between thecamera 105 and the actual mark (a simulated mark) corresponding to thecaptured mark.

The positions of the marks in the captured image may be indicated bypixel information of the position of each mark in the captured image,not distance information like XX (milimeters). For example, the pixelinformation includes the n-th pixel from an edge of the captured image.As a result, the camera system can calculate the distance from thecamera 105 to an actual object to be detected from the captured imageand the table data for distance detection.

The image processor 106 may derive an equation from the relation betweenthe positions of the marks in the captured image and an actual distance,and set the equation to the camera system. The calibration methodaccording to the first embodiment can produce a virtual situation that alot of marks are scattered over a vast area in the front of the camera105, by using two mirrors that face each other.

Therefore, taking a picture of the mirror by the camera 105 provides animage containing a lot of marks that scattered over the vast area. Thatis, the calibration method provides higher accuracy calibration for thecamera system in a small space. As a result, it is possible to calibratethe camera system with higher accuracy in a space limited such as afactory.

FIG. 6 is a schematic top view of a calibration apparatus according to asecond embodiment of the present invention. A calibration apparatus 200includes a calibration support device 210, cameras 105-1 and 105-2, andan image processor 206. The calibration support device 210 is differentfrom the calibration support device 110 of the first embodiment in thattwo marks are placed on the base 103. Since the arrangement of themirrors 101 and 102 is similar to that in the first embodiment.

In the calibration support device 210, two marks 104-1 and 104-2 areplaced between the mirrors 101 and 102, on the base 103. The marks 104-1and 104-2 may be placed at a position equidistant from the mirrors 101and 102 as in the first embodiment, or may be placed in other positions.In FIG. 6, the marks 104-1 and 104-2 are placed so that a lineconnecting between the marks is parallel to the reflective surfaces ofthe mirrors 101 and 102. The arrangement of the marks 104-1 and 104-2,however, is not limited to the arrangement shown in FIG. 6.

The image processor 206 has similar configuration as the image processor106 of the first embodiment, and the calibration processing (describedlater) is executed according to programs stored in the external storage.The calibration method according to the second embodiment is forcalibration of the stereo camera system that performs the planarprojection stereopsis method.

The planar projection stereopsis method refers to processes that includecapturing images by two cameras; associating all the pixels of the imagecaptured from one of the two cameras with points on a road; convertingthe pixels into a capturing view out of the other; and therebyrecognizing obstacles on the road.

According to this calibration method, the situation that a lot of marksare scattered over a vast area from around the camera is simulated, andthe result is captured as images by the cameras 105-1 and 105-2. First,the mirrors 101 and 102 are placed in predetermined positions from thecameras 105-1 and 105-2 (step Sb1 in FIG. 7). The mirrors 101 and 102are also arranged so that the reflective surfaces of the mirrors are inparallel and face each other.

After that, two or more marks are placed between the mirrors 101 and 102(step Sb2). The use of the calibration support device 210 makes stepsSb1 and Sb2 simply because of two mirrors and two marks, which are fixedin their positions. In other words, when the calibration support device210 is used, only the installation of the cameras 105-1 and 105-2remains. The positions (e.g., a distance between the two cameras) of twocameras 105-1 and 105-2 are fixed in the stereo camera system. Thepositions between two mirrors 101 and 102 and two cameras 105-1 and105-2 indicate such relations that the cameras 105-1 and 105-2 take apicture of a lot of marks (virtual marks) in one of the mirrors.

Adjusting the positions between the cameras 105-1 and 105-2 and thecalibration support device 210 is performed as follows: 1) move thestereo camera system while the calibration support device 210 is fixed;2) move the calibration support device 210 while the stereo camerasystem is fixed; or 3) move both the calibration support device 210 andthe stereo camera system. When the cameras 105-1 and 105-2 are fixed sothat the direction (the directions of the optical axes) that the cameraspoint shifts from the marks 104-1 and 104-2 as shown in FIG. 6, imagesas illustrated in FIGS. 8 and 9 is obtained.

FIG. 8 illustrates an image taken by the camera 105-1, and FIG. 9illustrates an image taken by the camera 105-2. Each of these includes alot of marks on an inclined line. The positions between the marks 104-1and 104-2 and the cameras 105-1 and 105-2 are not limited to those shownin FIG. 6, and may be determined depending on the purpose of the stereocamera system. For the calibration of the stereo camera system installedin the vehicle, the arrangement shown in FIG. 6 is preferable.

Once the positions of the cameras 105-1 and 105-2 and the calibrationsupport device 210 are adjusted, each of the cameras 105-1 and 105-2takes a picture (step Sb3). As a result, captured images containing alot of their virtual marks are obtained. The images obtained by thecameras 105-1 and 105-2 are input to the image processor 206.

The image processor 206 performs necessary work for the calibrationbased on instructions from the operator. The calibration includes twostages. The first stage is an adjustment of scopes of the cameras fortaking a picture and the captured images. The second stage iscalculations of parameters for the projective transformation withrespect to a plane.

First, the scopes of the cameras 105-1 and 105-2 are adjusted in thefirst stage (step Sb4). Ideally, the cameras 105-1 and 105-2 arepositioned so that their scopes overlap. The scopes of the cameras areadjusted by confirming the view from each camera. The adjustment is tochange the direction or the arrangement of the cameras so that a pointvanishing at infinity in a line of the mark 104-1 and their virtualmarks, and a point vanishing at infinity in a line of the mark 104-2 andtheir virtual marks, are positioned around the midpoint between sideedges of each image. As a result of the adjustment, it is preferablethat a position from the top edge of the image captured by the camera105-1 is the same as that of the image captured by the camera 105-2.

The images are adjusted in the next step of the first stage (step Sb5).In this step, if the images are distorted due to the characteristics ofthe cameras, the image processor 206 acquires parameters to correct thedistortion in the same manner as in the first embodiment. At the secondstage, the image processor 206 calculates parameters of the projectivetransformation with respect to a plane. The parameters are set to thestereo camera system (step Sb6).

First, an operator visually decides a relation between each mark in theimage captured by the camera 105-1 and each mark in the image capturedby the camera 105-2. It is necessary to include at least four pairs ofmarks when taking the decision. Each pair is specified using thecoordinates of the marks in the pair, and the coordinates are input tothe image processor 206. Each of the images captured by the cameras105-1 and 105-2 contains a line of the mark 104-1 and their reflectiveimage marks, and a line of the mark 104-2 and their reflective imagemarks, as shown in FIGS. 8 and 9. The relation is decided by comparingthe lines of the mark 104-1 between the two images and comparing thelines of the mark 104-2 between the two images.

The image processor 206 calculates the projective transformationparameters from the relation and the coordinates input. Suppose that apoint (ui, vi) in the image by the camera 105-1 corresponds to a point(ui′, vi′) in the image by the camera 105-2, and conversion parameter Hmeets the following equation (1):

$\begin{matrix}{{H = \left( {h_{11},h_{12},h_{13},h_{21},h_{22},h_{23},h_{31},h_{32},h_{33}} \right)}{u^{\prime} = \frac{{h_{11}u} + {h_{12}v} + h_{13}}{{h_{31}u} + {h_{32}v} + h_{33}}}{v^{\prime} = \frac{{h_{21}u} + {h_{22}v} + h_{23}}{{h_{31}u} + {h_{32}v} + h_{33}}}} & (1)\end{matrix}$where i is a natural number of not less than 1 and not more than N, andN>3.

When each coordinate value of the marks of N pairs is substituted intothe equation (1), simultaneous equations with an unknown value H isobtained. The projective transformation parameters are calculated fromthe obtained simultaneous equations, using, for example, the leastsquares method. It is necessary that the inequality N>3 be satisfiedbecause of the following reason. The conversion parameter H has nineparameters, and the eight of them are independent parameters. The numberof degrees of freedom, for example, is determined to eight by setting asum of squares of all parameters at one or setting h33 at 1.

The number of degrees of freedom of eight means all parameters arecalculated from eight equations. Since a pair of the marks derives twoequations for u and v, four pairs of the marks (i.e., pairs of more thanthree) is required to obtain eight equations. It is necessary that threeor more of these marks are not placed in a single line.

The image processor 206 stores the calculated projective transformationparameters in the parameter setting memory of the stereo camera system.If the cameras 105-1 and 105-2 have functions for distortion correctionof the image, the image processor 206 may store the transformationparameters for the distortion correction in the cameras 105-1 and 105-2.

According to a calibration method of the second embodiment, a situationthat a lot of marks are scattered over a vast area in front of thecameras 105-1 and 105-2 is simulated by using the two mirrors which faceeach other. Therefore, taking a picture of one of the mirrors by thecameras 105-1 and 105-2 provides an image containing a lot of marksscattered over a vast area. In other words, the captured imagecontaining a lot of marks widely scattered can be obtained even if thereis only a small space available.

Moreover, the calculation of the projective transformation parametersfrom the captured image allows more accurate projective transformationand thus the obstacle can be detected more accurately. The reason whythat it is preferable to scatter a lot of marks over a vast area will beexplained below.

The conversion parameter H is for directly converting the image by thecamera 105-1 into the image by the camera 105-2. This conversionactually is to project the image by the camera 105-1 in the plane in thereal space, and then to convert the projection result into the image bythe camera 105-2. Conversion that projects the image by the camera 105-1in a road plane in the real space is explained as an example to explainsimply. In addition, a relation between scattering in the direction ofthe depth of the mark (in a direction away from around the camera) andestimated errors of the parameters are explained based on the planarprojection stereopsis method.

In the planar projection stereopsis method, suppose that an optical axisof each camera is almost parallel in the direction of the depth and aspread in the horizontal and vertical directions is smaller than that inthe direction of the depth, a relation between coordinates (X, Y, Z) ofthe real space and coordinates (u, v) of the image is approximated bythe following equation:

$\begin{matrix}{\begin{pmatrix}u \\v\end{pmatrix} = {\frac{1}{Z}\begin{pmatrix}{p_{11}p_{12}p_{13}p_{14}} \\{p_{21}p_{22}p_{23}p_{24}}\end{pmatrix}\begin{pmatrix}X \\Y \\Z \\1\end{pmatrix}}} & (2)\end{matrix}$

The calibration of the camera is performed by calculating the value ofparameter p_(ij) by using n pairs of spatial coordinates (X_(i), Y_(i),Z_(i)) of the mark and coordinates (u_(i), v_(i)) of its image. In thiscase, i=1, 2 and j=1, 2, 3, 4.

Estimation of the parameters by least squares and its error will beexplained below. First, the following equation is obtained from equation(2):

$\begin{matrix}{{{Ap}_{1} = u}{{Ap}_{2} = v}{A = \begin{pmatrix}\frac{X_{1}}{Z_{1}} & \frac{X_{1}}{Z_{1}} & 1 & \frac{1}{Z_{1}} \\\; & \; & \vdots & \; \\\frac{X_{N}}{Z_{N}} & \frac{Y_{N}}{Z_{N}} & 1 & \frac{1}{Z_{N}}\end{pmatrix}}{p_{1} = \left( {p_{11},p_{12},p_{13},p_{14}} \right)^{T}}{p_{2} = \left( {p_{21},p_{22},p_{23},p_{24}} \right)^{T}}{u = \left( {u_{1},\ldots\mspace{14mu},u_{N}} \right)}{v = \left( {v_{1},\ldots\mspace{11mu},v_{N}} \right)}} & (3)\end{matrix}$

A least squares solution of equation (3) is given by:p ₁=(A ^(T) A)⁻¹ A ^(T) up ₂=(A ^(T) A)⁻¹ A ^(T) v  (4)

Set error variances of coordinates (u, v) of the image to σu2 and σv2respectively, suppose that there is no relation between the errorvariances, and the matrix Var is given by:

$\begin{matrix}\begin{matrix}{{{Var}(u)} = {\sigma_{u}^{2}\text{❘}}} \\{{{Var}(v)} = {\sigma_{v}^{2}\text{❘}}} \\{{{Var}\left( p_{1} \right)} = {{\sigma_{u}^{2}\left( {A^{T}A} \right)}^{- 1}A^{T}\text{❘}{A\left( {A^{T}A} \right)}^{- 1}}} \\{= {\sigma_{u}^{2}\left( {A^{T}A} \right)}^{- 1}} \\{= {\sigma_{u}^{2}{\sum\limits_{i}\;{\left( {\frac{X_{i}}{Z_{i}},\frac{Y_{i}}{Z_{i}},1,\frac{1}{Z_{i}}} \right)^{T}\;\left( {\frac{X_{i}}{Z_{i}},\frac{Y_{i}}{Z_{i}},1,\frac{1}{Z_{i}}} \right)}}}} \\{{{Var}\left( p_{2} \right)} = {{\sigma_{v}^{2}\left( {A^{T}A} \right)}^{- 1}A^{T}\text{❘}{A\left( {A^{T}A} \right)}^{- 1}}} \\{= {\sigma_{v}^{2}\left( {A^{T}A} \right)}^{- 1}} \\{= {\sigma_{v}^{2}{\sum\limits_{i}\;{\left( {\frac{X_{i}}{Z_{i}},\frac{Y_{i}}{Z_{i}},1,\frac{1}{Z_{i}}} \right)^{T}\;\left( {\frac{X_{i}}{Z_{i}},\frac{Y_{i}}{Z_{i}},1,\frac{1}{Z_{i}}} \right)}}}}\end{matrix} & (5)\end{matrix}$

Therefore, it is preferable that vector (X_(i)/Z_(i), Y_(i)/Z_(i), 1,1/Z_(i)) has scattered widely to reduce the error of the parameterp_(ij). From the fourth component of 1/Z_(i), especially, it ispreferable that the marks are distributed widely from near to far.Therefore, it is preferable for the captured image to contain the markswidely scattered in the direction of the depth to calculate theprojective transformation parameters.

In this respect, there are more marks in the second embodiment ascompared to those in the first embodiment. As a result, it is possibleto obtain a captured image containing a lot of marks scattered widelywithout preparing a vast space.

FIG. 10 is a schematic view of a calibration support device 310according to a third embodiment of the present invention. Thecalibration support device 310 is different from the calibration supportdevice 110 of the first embodiment in that a hole 311 through which thecamera takes a picture is provided in a mirror 302. The hole 311 has anaperture whose size is approximately the same as the lens aperture ofthe camera 105. Moreover, the hole 311 has a shape that does not hinderthe view of the camera 105.

FIG. 11 is an illustration indicating a relation between an optical axis602 of the camera 105 and an upper bound line 603 to obtain thereflective image in the first embodiment. To take a picture of thereflective images which simulate the marks positioned remotely, thecamera 105 needs to capture the images near the optical axis 602, asshown in FIG. 11.

In the first embodiment, however, since the camera 105 takes a pictureof the mark 104 and its reflective images over the mirror 102, themirror 101 is not in the optical axis 602 of the camera 105. Therefore,scenery in the back of the mirror 101 is included in the captured image,and thus it is impossible to obtain the reflected images that simulateonly the marks.

On the contrary, in the third embodiment, since the mirror 302 has thehole 311 and the camera 105 is installed so that the optical axis 602passes through the hole 311, it is possible to obtain the reflectedimages that simulate only the marks. A transparent material (e.g.,glass, plastic, or acrylic fiber) or translucent materials (e.g., halfmirror) may be filled in the hole 311. Alternatively, a notch is formedin the mirror 302, instead of the hole 311. In other words, the mirror302 may be provided with not the hole 311 but an optical transmissionmember that allows light to pass from the back (the mirror 101 and theopposite side) of the mirror 302 to the mirror 101.

Though only one camera is shown in FIG. 10, when the calibration supportdevice 310 is used for the calibration of the stereo camera system thatprovides with at least two cameras like in the second embodiment, themirror 302 may be provided with optical transmission members forrespective cameras. In this case, the mirror 302 may be provided with asingle optical transmission member, and may alternatively be providedwith a plurality of optical transmission members corresponding to theposition of each camera.

According to a calibration method of the third embodiment, it ispossible to capture the reflected images that simulate only the marks.Therefore, it is possible to obtain the captured image containing a lotof marks widely scattered without preparing a vast space, and thus tocalibrate the camera system with high accuracy.

In the third embodiment, the mirror 101 may be provided with the opticaltransmission member. The camera 105 is fixed behind the mirror 101, andtakes a picture of the reflective surface of the mirror 302 when themirror 101 is provided with the optical transmission member.

FIG. 12 is a schematic view of a calibration support device according toa fourth embodiment. The calibration support device 410 of the fourthembodiment is different from that of the first embodiment in that a halfmirror 402 is used in place of the mirror 102. In the first embodiment,the camera 105 is fixed above the mirror 102. On the other hand, in thethird embodiment, the camera 105 is fixed in a place corresponding tothe position where the hole 311 is installed.

In the fourth embodiment, the camera 105 is fixed so that the opticalaxis of the camera 105 passes through the half mirror 402. As a result,the camera 105 can capture the reflected image in a direction almostparallel to the optical axis of the camera 105.

FIG. 13 is an illustration indicating a relation between the opticalaxis 602 of the camera 105 and an upper bound line 603. Although theoptical axis 602 and the upper bound line 603 overlap in the figure, itis possible to obtain the reflected images in a direction very near theoptical axis 602. Although a single camera is shown in FIG. 12, thecalibration support device 410 may be used for the calibration of thesystem that includes a plurality of cameras.

According to the fourth embodiment, it is possible to take a picture ofthe image containing the reflected images that simulate the marksremotely, by the camera installed behind the half mirror 402.

The present invention is not limited to the embodiments as describedabove, and may be provided as various modifications explained below. Thecamera system may be manufactured using any of the calibration methodsdescribed above. For the manufacturing process including the process ofthe calibration method, manufacturing the camera system (a system usedfor the obstacle detection device, a vehicle driving support device, ora vehicle autopilot) installed in the vehicle will be explained below.

As shown in FIG. 14, the camera of the camera system acquired bymanufacturing or bought by a user is installed in a predeterminedposition (step Sc1). The predetermined position is, for example, theupper part of a dashboard near the windshield, or the back of therearview mirror (the reflective surface and the other side), of thevehicle. It is assumed that the camera system is composed of the cameraand the main body. The main body includes a setting memory where data(e.g., correction parameters, the projective transformation parameters,and the table data) obtained by the calibration is set.

After the camera is installed in the predetermined position, thecalibration according to one of the embodiments as described above isperformed (step Sc2). For example, when the calibration method accordingto the second embodiment is performed, the camera installation positionis fine-tuned and the correction parameters are calculated. Moreover,the projective transformation parameters are calculated, and the resultis stored in the setting memory.

As a result, it is possible to manufacture the camera system that theparameters for position detection with higher accuracy are set,according to this manufacturing method including the calibration.

According to the first to fourth embodiments, the camera to becalibrated is installed behind the mirror 102. For example, in the firstembodiment, the camera 105 is adjusted so as to point to the reflectivesurface of the mirror 101 and it is fixed above the mirror 102. If it iseasy to carry the camera system, the adjustment of the camera and thecalibration support device is also easy.

In the camera system (e.g., a system for the obstacle detection device)installed in the vehicle, however, it is difficult that the calibrationsupport device is set up in front of the camera. For example, when thecalibration is performed in a production line for the vehicle, since thevehicle is conveyed in the same direction as the front of the vehicle,the calibration support device set up in front of the camera interruptsthe manufacturing operation of the vehicle.

To solve this problem, the mirror 901 is placed in front of the camera105 to refract the optical axis 602 of the camera 105 as shown in FIG.15. The calibration support device of each of the embodiments is set upin the optical axis 602 refracted by the mirror 901. That is, the mirror901 is placed in front of the camera 105 so that the image whichreflects in the mirror 901 from the position of the camera 105 coincideswith a forward view out of the camera.

As a result, the camera 105 can take a picture of an image containing alot of marks. Therefore, when the calibration is to be performed, themirror 901 is moved forward of the camera.

If the calibration support device is set up sideward of the productionline in the factory, the captured image containing a lot of marks can beobtained. Namely, the manufacturing operation of the vehicle is notinterrupted. Even when the calibration of the stereo camera system witha plurality of cameras is performed, the mirror 901 is placed in frontof the camera, so that it is possible to take a picture of the imagecontaining a lot of marks in the reflective surface of the mirror 101that reflects in the mirror 901.

The vehicle carried on the production line is generally elevated thanwhen the vehicle is on the road. Therefore, when a reference plane ofthe calibration of the camera system installed in the vehicle is a floorof the factory, the calibration is performed with respect to a positionlower than the road where the vehicle runs. However, since the camerasystem such as the obstacle detection devices installed in the vehicledetects a distance based on a mark such as an object (a white line) onthe road, it is desirable that a mark for the calibration is placed atthe same level as the road.

It is preferable to match a plane level at which the mark for thecalibration is placed to a reference plane used when the camera systemactually detects a distance, that is, a level of a reference plane usedfor the processing performed by the camera system. Therefore, when thecalibration is performed in the manufacturing process of the vehicle,the calibration support device may be constituted as shown in FIG. 15.

FIG. 15 is an example of the calibration support device when thecalibration of the camera system is performed in the manufacturing ofthe vehicle. This calibration support device includes a base 103-2, themirrors 101 and 102, and a mark (not shown in the figure) on the base103-2.

The calibration support device also includes level adjustors 505 toadjust height of the base 103-2 from a floor 910. A vehicle 902 is at aposition higher than the floor 910 since the vehicle 902 is carried onthe production line. The plane on which the vehicle 902 is placed is avirtual ground plane 103-1. For the vehicle 902, the virtual groundplane 103-1 is a plane at the same level as the road.

A camera system 1501 installed in the vehicle 902 uses a road surface asa reference plane for actual processing of, for example, obstacledetection. That is, the level of the reference plane is the same as thevirtual ground plane 103-1. Therefore, it is preferable that the markfor the calibration is placed at the same level as the virtual groundplane 103-1.

The level adjustor 505 adjusts so that the base 103-2 of the calibrationsupport device is at the same level as the virtual ground plane 103-1.As a result, the mark for the calibration is placed at the same level asthe virtual ground plane 103-1. Since the level at which the mark isplaced only has to coincide with the level of the road, the mark may beplaced at the same level as the virtual ground plane 103-1. For example,the top of a pole that has the same height as a height between the floor902 and the virtual ground plane 103-1 is marked up.

The calibration support device may include a mirror 1002 to refract theoptical axis of the camera 105 installed in a vehicle 1004 upward, asshown in FIG. 16. As a result, the camera 105 can take a picture of theimage containing a lot of marks that reflect in the mirror 101 installedabove the vehicle. In other words, only since the mirror 1002 is movedforward of the camera, the captured image containing a lot of marks isobtained even in the factory where the calibration support device is setup.

Calibration based on the road can be performed for this example byplacing the mark 104 in a virtual plane (middle point line in FIG. 16)including a virtual ground plane 1003-1 of camera 105. This virtualplane is a plane where the virtual ground plane 1003-1 is refracted bythe mirror 1002 and extends to the mirror 101. Therefore, this virtualplane includes the virtual ground plane 1003-1 and a mark arrangementplane 1003-2 where the mark 104 is placed. That is, a virtual plane isthe result of combining the virtual ground plane 1003-1 and the markarrangement plane 1003-2, and has a shape that these planes intersect ata position of the mirror 1002.

In FIGS. 15 and 16, although the optical axis of the camera is refractedonly once by a single mirror (the mirror 902 or the mirror 1002), theoptical axis may be refracted a plurality of times by a plurality ofmirrors. For example, when the projective transformation parameters arecalculated, it is necessary to check the relations between two imagescaptured by the stereo camera. When the relations are checked, it isdesirable that the right and left positions of the marks of the imagesare recorded. Therefore, it is preferable that another mirror isprepared and the optical axis is refracted two times.

As shown in FIG. 17, the calibration support device includes a mirror1101 placed forward of the cameras 105-1 and 105-2 fixed in a vehicle1104 and a mirror 1102 placed symmetrically with respect to the mirror1101. As a result, optical axes 601-1 and 601-2 of respective camerasare substantially refracted.

Moreover, although the mark 104 (a mark 104-1 and a mark 104-2) has apoint-shape (almost circular in the figure) in each of the embodiments,the mark may be a line that is normal to the reflective surfaces of themirrors 101 and 102. If the mark is a line, a straight reflected imagein addition to the line mark is projected in the mirror 101 (or themirror 102). Therefore, a situation as if a white line extends from nearthe camera to far on the road is produced. As a result, when in thecalibration it is assumed that a white line is only at one side of theroad lane, even if the system uses two cameras as described in thesecond embodiment the calibration support device only has to includeonly one of the marks 104-1 and 104-2.

In each of the embodiments, the calibration apparatus includes the imageprocessor (106, 206). This image processor performs the settingprocessing for calculating the projective transformation parameters andthe like. When the camera system to be calibrated has functions for thecalibration, the camera system may perform the calibration processingsuch as setting for calculating parameters and the like based on theimage captured by the camera.

Additional advantages and modifications will readily occur to thoseskilled in the art. Therefore, the invention in its broader aspects isnot limited to the specific details and representative embodiments shownand described herein. Accordingly, various modifications may be madewithout departing from the spirit or scope of the general inventiveconcept as defined by the appended claims and their equivalents.

1. A method for calibration of a stereo camera system that uses aplurality of cameras, comprising; disposing a first mirror and a secondmirror in such a manner that reflective surfaces of the first mirror andthe second mirror face toward each other; placing at least two marks ona reference plane between the reflective surface of the first mirror andthe reflective surface of the second mirror so that a plurality ofreflected figures of the each mark are generated by the first mirror andthe second mirror, the reference plane used in processes performed bythe stereo camera system; capturing with the cameras an image includingthe plurality of the reflected figures of the each mark; determiningposition of the reflected figures in each of the images, wherein theposition of each of the reflected figures is represented by a pair ofcoordinates; and calculating projective transformation parameters forplanar projection stereopsis method by using the position of thereflected figures.
 2. The method according to claim 1, wherein the firstmirror is a half mirror.
 3. The method according to claim 1, furthercomprising: positioning the marks at the same height as that of thereference plane.
 4. The method according to claim 1, wherein the firstmirror has a portion that passes light.
 5. The method according to claim1, wherein the marks includes two marks that are disposed on a lineparallel to the reflective surfaces of the first mirror and the secondmirror.
 6. The method according to claim 1, wherein each of the marks isa line that is normal to the reflective surfaces of the first mirror andthe second mirror.
 7. The method according to claim 1, furthercomprising: disposing a third mirror in such a manner that the marksreflected in the first mirror and the second mirror are also reflectedin the third mirror, wherein images reflected in the third mirror arecaptured with the camera.
 8. An apparatus for calibration of a stereocamera system that uses a plurality of cameras, comprising; a firstmirror having a reflective surface; a second mirror having a reflectivesurface, wherein the reflective surfaces of the first mirror and thesecond mirror face toward each other; a plurality of marks that aredisposed on a reference plane between the reflective surface of thefirst mirror and the reflective surface of the second mirror so that aplurality of reflected figures of the each mark are generated by thefirst mirror and the second mirror, the reference plane used inprocesses performed by the stereo camera system; and an image processorthat calculates projective transformation parameters for planarprojection stereopsis method by using position of reflected figures ofthe marks in images captured by the cameras, wherein the position ofeach of the marks is represented by a pair of coordinates.
 9. Theapparatus according to claim 8, wherein the first mirror is a halfmirror.
 10. The apparatus according to claim 8, further comprising: alevel adjustor that positions the marks at the same height as that ofthe reference plane.
 11. The apparatus according to claim 8, wherein thefirst mirror has a portion that passes light.
 12. The apparatusaccording to claim 8, wherein the marks includes two marks that aredisposed on a line parallel to the reflective surfaces of the firstmirror and the second mirror.
 13. The apparatus according to claim 8,wherein each of the marks is a line that is normal to the reflectivesurfaces of the first mirror and the second mirror.
 14. The apparatusaccording to claim 8, further comprising: a third mirror disposed insuch a manner that the marks reflected in the first mirror and thesecond mirror are also reflected in the third mirror; wherein imagesreflected in the third mirror are captured, with the camera.
 15. Amethod of manufacturing a stereo camera system that uses a plurality ofcameras and a memory, comprising: installing the cameras in apredetermined position; disposing a first mirror and a second mirror insuch a manner that reflective surfaces of the first mirror and thesecond mirror face toward each other; placing at least two marks on areference plane between the reflective surface of the first mirror andthe reflective surface of the second mirror so that a plurality ofreflected figures of the each mark are generated by the first mirror andthe second mirror, the reference plane used in processes performed bythe stereo camera system; capturing with the cameras an image includingthe plurality of the reflected figures of the each mark; determiningposition of the reflected figures in each of the images, wherein theposition of each of the reflected figures is represented by a pair ofcoordinates; calculating projective transformation parameters for planarprojection stereopsis method by using the position of reflected figures;and storing the projective transformation parameters in the memory. 16.The method according to claim 15, wherein the first mirror is a halfmirror.
 17. The method according to claim 15, further comprising:positioning the marks at the same height as that of the reference plane.18. The method according to claim 15, wherein the first mirror has aportion that passes light.
 19. The method according to claim 15, whereinthe marks includes two marks that are disposed on a line parallel to thereflective surfaces of the first mirror and the second mirror.
 20. Themethod according to claim 15, wherein each of the marks is a line thatis normal to the reflective surfaces of the first mirror and the secondmirror.
 21. The method according to claim 15, further comprising;disposing a third mirror in such a manner that the marks reflected inthe first mirror and the second mirror are also reflected in the thirdmirror, wherein images reflected in the third mirror are captured withthe camera.